Video processing for composite images

ABSTRACT

The image composition system includes framestores 30, 31 for receiving information from first and second picture sources. A processor 33 provides the composed image by using information from these sources. The processor is controlled by picture shape information made available from a third framestore 32. This shape information may be provided by a camera 26 receiving an image of a silhouette for example or the shape can be manually generated via a touch tablet 38. The instantaneous value of the shape controls the blending of the pictures such that portions of the picture can be taken from a scene and inserted without noticeable degredation. Manipulation of the position, orientation or size of the inserted picture portion for example can also be effected.

BACKGROUND TO THE INVENTION

The invention relates to image composition arrangements. The compositionof pictures where one image is keyed into a second picture is knownusing simple switching techniques.

Unfortunately, the results are not altogether visually satisfactory andthe manipulation of such images is limited.

The present invention is concerned with providing an arrangement whichallows a more realistic resultant composed image to be provided andwhich provides greater flexibility than heretofor.

SUMMARY OF THE INVENTION

According to the invention there is provided an image composition systemcomprising, first input means for providing a first source of pictureinformation, second input means for providing a second source of pictureinformation, means for providing a third source of picture information,processing means for receiving information from both said first andsecond sources so as to provide the composed picture, and control meansfor varying the proportions of picture information used by saidprocessing means to provide said composed image in dependence on theinformation from said third picture source.

Further according to the invention there is provided a method for imagecomposition comprising, receiving a first source of picture information,receiving a second source of picture information, receiving a thirdsource of picture information, processing the information from both saidfirst and second sources by variably controlling the proportions ofpicture information used in the processing step in dependence oninformation from said third picture source.

BRIEF DESCRIPTION OF THE DRAWINGS

The first and/or second information sources may comprise a picturesource, color information source or synthetically generated source.

The invention will now be described by way of example with reference tothe accompanying drawings, in which:

FIG. 1 shows one embodiment of the present invention for composing animage produced from more than one image source,

FIG. 2 shows visual representation of the image composition,

FIG. 3 shows various parameter values illustrating the blendingtechnique employed in the image compsition,

FIG. 4 shows an embodiment of the processor of FIG. 1,

FIG. 5 shows additional manipulation components,

FIG. 6 shows an arrangement allowing the capability of moving theinserted picture,

FIG. 7 shows an arrangement for artificially generating the insertshape,

FIG. 8 shows an arrangement allowing the insert to be transferred to theoriginal picture, and

FIG. 9 shows an arrangement for providing a mask to aid picture buildup.

DESCRIPTION OF PREFERRED EMBODIMENTS

As already discussed, established composite image generation techniquestend to produce unrealistic results which appear contrived or degraded.This degradation can be more pronounced when the data is in digitalformat, due to the quantized nature of a digital television picture. Toachieve enhanced results the present invention is concerned withmanipulating the picture information in such a way that the composedpicture is a composite picture from more than one source blended in amanner which visually results in a produced image which is generallyindistinguishable from images which were composed originally as a singlepicture yet allowing manipulation of the composition of this picture tobe effected.

FIG. 1 shows one embodiment of the system of the present invention forproducing the image composition. The first picture source is provided bycamera 20 and passes via analogue to digital converter (ADC) 27 to framestore 30.

The second picture source is provided by camera 21 and passes via ADC 28to framestore 31. The outputs from framestores 30 and 31 are madeavailable to processor 33 described, in more detail below, and theresult therefrom is available for display on a monitor 34, via DAC 39 asnecessary, or for use elsewhere in analogue or digital form as desired.Thus the composed image from processor 33 can be considered as beingcomprised of picture information from both original picture sources. Theway in which these sources are used within the processor is effectivelycontrolled by a third source of picture information. This is shown asbeing provided by the additional framestore 32.

This further framestore 32 contains picture shape and blendinginformation for use in controlling the processor 33. This informationmay have been derived via camera 26 and ADC 29 or touch tablet 38 asexplained below and can be considered as a variable stencil for use incomposing the picture.

The resultant manipulation is exemplified in FIG. 2. The first andsecond images within framestores 30 and 31 respectively are received byprocessor 33. The control image from framestore 32 is used to cause theprocessor to compose the final image from the first image together withthat portion of the second image corresponding to the shape of thecontrol image. This allows only selected features from within the secondimage to be used in the final image. Thus the person can be transportedfrom the original indoor scene shown into an outdoor scene as shown onmonitor 34. In practice the processor is also configured to manipulatethe data in such a way that the insert is realistically blended into thepicture to appear as if it were there in the original scene. The controlimage itself is the mechanism which directs this blending operation,using both its shape and instantaneous value as now explained.

The control image is arranged to effectively define the percentage usedfrom one picture when processed with the other picture, with blendingfor example adjacent the picture insert interface. This value (K) isshown in the example of FIG. 3c as varying initially from a minimum to amaximum adjacent boundary I and then subsequently decreasing to aminimum adjacent boundary II for that T.V. line. In the FIG. 2 example,this could correspond to a T.V. line towards the bottom of the frame. Atthe changeover, this technique avoids any sharp edges by providing agradual increase in picture contribution spaced over one or more picturepoints. Thus, adjacent the first boundary, a small contribution is madefrom the picture about to be inserted (picture B) and this increases,with a corresponding decrease in picture A until B completely replacesfirst picture source A. When the next boundary is approached, theoperation repeats, this time in reverse. This technique results inblending of the pictures from the first and second sources in thisexample only in the marginal regions of their interface. Although theblending described can be considered as along a horizontal line, thesame technique is employed vertically. At other parts of the picture therelationship will be different. Since the transition point may bedisplaced on subsequent scan lines, each line will result in slightlydifferent values of K as exemplified in FIG. 3d. For any parts of thepicture where no contribution is required from the second picture sourceB, then K will be a minimum throughout the horizontal scanning line (seeFIG. 3a). Where the insert has a horizontal edge, then adjacent thisboundary, a value of K shown in FIG. 3b could be expected for therelevant scanning line. Adjacent lines would have an increasing value ofK until the FIG. 3c situation was reached, thus giving the blendingtechnique vertically as well as horizontally.

The frame stores 30 to 32 share common write/read addressing block 35,which is under the control of the input sync generator 36 and outputsync generator 37 in normal manner.

The process within processor 33 required to achieve the blending isgiven by the equation:

    OUTPUT=K×PICTURE 1+(1-K)×PICTURE 2

where K<1.

An embodiment for the processor 33 is shown in FIG. 4. The value of Kfrom control shape store 32 for a given pixel location is received bymultiplier 41 and its inverse (1-K) is made available to multiplier 40via inverter 43 to control the image composition. The outputs frommultipliers 40, 41 are received by adder 42, the output of which can bepassed to the monitor as described in FIG. 1.

Although the system is shown for simplicity as having single framestores 30 and 31 for handling monochrome only it can be extended tocolor by adding additional memory to these picture framestores and alsoadding parallel processing circuitry for the color data.

Although the picture shape with associated values of K for each picturepoint within the frame for store 32 could be generated synthetically, apreferable way of providing these values is to use a visual shapemechanism. One approach shown in FIG. 1 is to use a camera 26 with itsoutput passed to store 32 via ADC 29. The insert shape can be a profileor silhouette which will in practice cause a slope to be produced in theanalogue output level from the camera over a number of picture points asin FIG. 3c and thus when digitized and stored as 8 bit words for examplewill give the variation in K desired for smooth blending in thechangeover region.

By providing shapes with intermediate intensity values for K throughoutthe insert, we have found that special effects such as transparent ortranslucent images for example can be included in the composed scene.

By incorporating a high resolution camera or by including filteringtechniques, the number of pixels involved at changeover (horizontallyand vertically) and thus the gradient of the slope can be varied.Another approach shown in FIG. 1 is to use a touch tablet 38 or othermanually operable device to provide the desired insert shape and usethis as the k data input to the store 32 using techniques extending fromthese described in UK Patent Publication No. 2089625 for example, asdescribed with reference to FIG. 9 below.

Although the picture inputs to the stores 30 and 31 have been describedas coming from cameras, the system is flexible enough to allow otherpicture sources to be used. One specific aspect is to compose a picturecontaining graphic information. In this case store 30 can provide thebackground information (luminance or color) and the graphic shape can beinput to store 32 as before. These shapes could be derived from anysuitable source but for realism the manner described in FIG. 9 below ispreferable. The shape could be a straight line, circle, alpha/numeric orother character if desired. In the present situation the store 31 couldmerely contain a fixed (or variable) color or intensity which would beselected dependent on shape defined by store 32.

Now wherever the store 32 gives a value of zero then the output fromframestore 30 will be passed to the monitor 34 without modification, butwhen the store 32 output equals one then the color as defined by thestore 31 will appear on the monitor. For values between nought and one aproportion of mix of color value and framestore 30 output will beapplied to the monitor.

Where the system is used in conjunction with the image creation systemmentioned above the picture visible on the monitor directly simulatesthe effect which the artist will achieve on his picture when he finallydecides to commit these lines and other graphic representations to hispicture, or alternatively can use these lines as a guide line as thepicture is created.

Although the FIG. 1 arrangement is shown with common addressing forframestores 30-32, so that there is a fixed pixel relationship betweenthe images stores therein, an additional benefit can be achieved as nowdescribed with reference to FIG. 5 to allow a changeable pixelrelationship with additional manipulation so that, whilst retaining theoriginal picture information, is is possible to move the location, sizeor orientation of the insert into the composed picture.

The FIG. 5 arrangement is concerned with the manipulation of informationfrom the framestores and, for simplicity, only the relevant blocks areincluded. The input and output arrangements would be as in FIG. 1. Theaddress generator 35 is now used only with the first picture framestore30. An additional address generator 44 is provided for use by both thesecond picture framestore 31 and the control shape image framestore 32.The outputs from framestores 31 and 32 now pass via interpolators 47 and48 prior to receipt by processor 33. The read addressing of theframestores and the control of the interpolators are effected bymanipulator 49 to give the required address selection and pixelinterpolation to achieve the size change or orientation desired forexample. The mechanisms for interpolation and address manipulation areknown in the art see also U.S. Pat. No. 4,163,249 for example. Becausethe addressing block 44 is common to framestores 31 and 32 the pixelrelationship is maintained. This ensures that the manipulation of thecontrol image shape is duplicated for the image within framestore 31.Thus, considering the images represented in FIG. 2, the control imagecan be manipulated so that it can be shrunk for example and the personin the second image will shrink also and be inserted in the picture atreduced size. Rotation manipulation of the control shape will cause theperson to lie down in the final image for example.

The requirement to produce a new shape representation each time isremoved. Manual control of the position, size or orientation can beachieved using trackerball or joystock for example in usual manner.

Where there is a requirement to only move the control image location analternative system can be used without employing interpolation as nowdescribed.

An example of the use of this technique is for `cut and paste`. Anartist often finds that he is painting the same pictures over again andit would greatly assist him if he could have a means of taking part ofthe previously drawn picture and pasting this into his new picture.

Prior to pasting together his pictures the artist may require that thepicture to be pasted can be moved around the original screen and viewedas if it existed in that picture but without finally committing it tothe picture, this can be achieved using a system as shown in FIG. 6. Thepicture framestores 30, 31 now each has its own address generator 35, 44which can be independently controlled. If the start address of generator44 is varied, while the start address of address generator 35 is kept at0.0, then using the shape as defined in store 32, the picture cut frompicture B can be made to move around the picture A until its desiredplacement is found. This movement can be controlled by joystick, trackerball or other means.

In addition a further shape store 45 is provided as a refinement toallow a foreground object to be defined in picture A. This store 45 isalso driven by address generator 35. This store contains the shape orblending information which defines the foreground object in picture A.The processor 33 has been modified to include a further multiplier 46 tocope with the additional manipulation. To produce the correct K factorfor the processor the outputs from stores 45 and 32 are first multipliedtogether before being applied to the processor as previously described.The result as far as the artist or operator is concerned will be that ashe moves his cut picture from picture B around picture A then it willseem to disappear behind the objects in picture A which are defined asbeing foreground.

The system is shown with the capability of scrolling both or eitherpicture A or B using addressing blocks 35 or 44.

The common address generator 35 for stores 30 and 45 and the commonaddress generator 44 for stores 31 and 32 ensures that correct picturerelationships are maintained.

The ability to move and insert parts of the picture make the systemideal for providing animation. An image of an animal or parts thereofcan be inserted and moved across the screen and if a separate cut outshape of the legs are stored then these can be manipulated and capturedframe by frame under joystick control for example to simulate a walkingmovement whilst the animal tranverses the picture.

Alternatively, the positions can be generated under computer control,where a range of repetitive movements can be made available and socreate simply and less tediously animation effects.

As already mentioned a touch tablet 38 or its equivalent can be used toenter the desired shape information into stores 32 or 45. In order toproduce the blending effect desired, our research has shown that thiscan be provided by utilizing techniques derived from modifiedarrangements to those disclosed in UK Patent Publication No. 2089625.Thus the touch tablet 38 of FIG. 1 is in practice used with additionalsystem elements to provide the desired painting or drawing techniques asnow described in detail in FIG. 7.

The co-ordinates from the touch tablet 38 are received by an addressgenerator 53 which provides addressing for the frame store 32 so as tocorrectly enter the data into the store locations. The frame store couldalternatively be store 45 of FIG. 6. The address generator controls theframe store to allow a `read-modify-write` sequence to occue, themodification taking place in processor 50. The address generator 53 alsocontrols the stores 51 and 52 which have a size corresponding to adesignated number of picture points in a patch. The pencil intensity (orcolor) and pencil shape provided by stores 51 and 52 are an ideal way ofproviding the insert shape as the artist or operator can draw around theobject of interest and then fill in the space inside the outline.Because of the pencil shape being of the type that falls away at theedges, this will also cause the desired blending effect as nowdescribed.

A patch of 16×16 pixels is shown as being large enough to encompass thedesired pen shape in this example. The peak of the pen shape is centralof the patch in this instance and will produce the maximum value of K atthis point. The x and y coordinate provided by the touch tablet willcorrespond to the corner of the patch read out from the store andprocessing of all points within this patch is effected in processor 50and the modified data written back into the store 32 (or 45). Duringthis processing the old luminance value and the designated intensityvalue are subtracted in subtractor 55 and the difference multiplied bycoefficient K in multiplier 56, the value of K being dependent on wherethe particular picture point lies within the selected patch. The resultis added to the earlier luminance data by adder 57. It is clear thatsome picture points at the periphery will remain unchanged in thisexample. Movement of the actual stylus on the touch pad by one picturepoint will cause a new patch to be read out from the store 32 which willcontain most of the earlier picture points but 16 new pictures will bepresent and naturally 16 others will have been omitted. The processingwill again be carried out for the entire patch. It can be seen thatduring the second processing operation just described, the previousmovement by 1 picture point will cause a proportion of the luminanceinformation generated by the earlier processing operation to be used inthe calculation of the new content for the updated patch.

The number of processing steps for a given coordinate will depend on thesize of the patch accessed.

Thus, if the patch was say 32 picture points wide and 32 high there are32×32 or 1024 points to be processed for each movement of the stylus.

In this way, the insert shape is built up. At the edges the intensitywill automatically fade away, allowing the blending effect to beachieved as the desired value of K will have been provided and enteredinto stores 32 or 45 during this operation. Shapes drawn with variableintensity other than at the boundary will cause variable blendingelsewhere. The operator can view this shape during or after generationby feeding the data from the relevant store to the monitor 34, or byconsidering it as a graphic input and superimposing it on the originalpicture as described above.

Once the operator or artist has decided where he requires his cutpicture to be placed then there is a need for actually transferring thecut picture from picture B to picture A. Once again, the cut picturemust be blended in properly when interfaced to the original picture andthis can be achieved using the arrangement shown in FIG. 8. In thiscase, however, instead of the pencil color and pencil shape togetherwith framestore 32 output being applied to the processor 50 as in FIG.7, the framestore 31 plus its insert shape as defined by store 32 areapplied to the processor in place of the pencil or the implement.

As for the video path this processing path also requires that framestore30 and framestore 31 have separate address generators 35, 39 as theyhave to be accessed from different addresses for a particular process.To achieve the foreground/background effect then the outputs from thestores 32 and 45 are multiplied in an additional multiplier 62 beforebeing received by the other elements of the processor. A furthermultiplier 60 is provided which acts as an input for pressure providedfrom transducer 61 thus allowing the information from framestore 31 tobe only partially applied onto the framestore 30 information if sodesired.

A dedicated processor is required for block 50 in order to achievereasonable processing speed. A full framestore of 584×768 pixels takesabout 0.6 of a second to be pasted into the first framestore 30. Sincein many cases the object to be cut from framestore 31 does not occupythe whole of this framestore then a saving in time can be made by onlyaccessing a rectangle which is sufficient to enclose the object to becut. This patch size can be controlled by a patch size selector 63 asshown which in turn controls the implementing of the two addressgenerators. In practice this patch size generator could incorporate asimilar addressing mechanism to that used for the address generationwithin block 53 of FIG. 7 and described in more detail in theaforementioned patent application.

The output provided by this processing is given by the following:##EQU1## where K₁ is the output provided by store 45

K₂ is the output provided by store 32

FS₁ is the output provided by store 30

FS₂ is the output provided by store 31.

Although the control image shape and intensity has been described asbeing generated in a single operation, in practice the shape can befirst defined and the intensity or color built up subsequently. Thisallows special effects mentioned above to be readily altered by theoperator.

From the artistic point of view this is similar to applying a form ofmasking tape or stencil to the picture so defining the area within whichthe artist requires his paint applied to the paper. This is ofparticular use when using an airbrush but may also be used for any otherpainting media. This arrangement now described provides a means ofproducing the equivalent of painting mask electronically as shown inFIG. 9. The processor is similar to that of FIGS. 7 and 8 and the systemincludes the store 32 which is driven in parallel with the framestore30. When the pencil color (or intensity), pencil shape and framestorevalue are read out an additional value is read out from the store 32which defines the mask. This is multiplied in multiplier 62 by thepressure from the stylus via transducer 61 before the further processorsteps. Thus the store 32 modulates the pressure such that when the maskis one the pressure is allowed through and there is no effect upon thepainting but when the mask is zero then the pressure is turned to zeroand no painting will appear on the framestore. Since the mask can haveany value between nought and one a shape can be applied to it which willdefine the exact shape in which the artist requires his paint to beapplied.

Once again, the mask provides a blend of the required paint to theoriginal picture and so producing a very natural effect.

Thus the various embodiments described allow a composite picture to begenerated from normal picture sources or by synthesis, which retains itsrealism by removing sharp edges from the interfaces and allowingadditional manipulation such as relative movement to be provided, underthe control of the image shape.

We claim:
 1. An image composition system comprising:a first input forfirst digital image signals representing an image; first frame storemeans for storing a frame of said first digital image signals; a secondinput for second digital image signals representing an image; secondframe store means for storing a frame of said second digital imagesignals, a third input for first digital control image signalsrepresenting a control image; a third frame store means for storing aframe of said first digital control image signals; a fourth input fordigital image signals representing a second control image; fourth framestore for storing a frame of said second control image signals; andcombining means for producing a composed image by selecting for saidcomposed image only signals from said first frame store means when saidfirst control image signals have a particular value and for selectingfor said composed image signals from one or the other or both of saidfirst and second frame store means, depending on the value of saidsecond control image signals, when said first control image signals havea different value, and for combining any image signals selected fromboth of said first and second frame store means in order to produce saidcomposed signal.
 2. An image composition system according to claim 1,wherein said first frame store means and said third frame store meanshave separately controllable address generators to enable the image inone store to be modified spatially relative to that in the other.
 3. Animage composition system as in claim 2 comprising operator controlledgraphic means arranged to provide the digital control image signalsrepresenting a control image to a selected one of said third and fourthinputs.
 4. An image composition system as in claim 3 wherein saidgraphic means include a manually operable coordinate selecting means anda processor for synthesizing the control image in dependence uponcoordinates made available by said selecting means.
 5. An imagecomposition system as in claim 1 comprising operator controlled graphicmeans arranged to provide the digital control image signals representinga control image to a selected one of said third and fourth inputs.
 6. Animage composition system comprising:first input means for providing afirst source of picture information representing a first image; secondinput means for providing a second source of picture informationrepresenting a second image; means for providing a third source ofpicture information representing a control image; manipulating means forselectively manipulating the picture information of said second andthird sources to provide a common change in the spatial relationship ofthe respective image and control image from the second and third sourcesrelative to the first image; processing means for receiving informationfrom said first source, and from said second source after manipulation,if any, by said manipulating means, and for providing therefrom acomposed image; and said processing means including control meansresponsive to said control image for varying the proportions of pictureinformation from said first and second sources used by said processingmeans to provide said composed image, in dependence on the informationfrom said third source.
 7. An image composition system comprising:afirst input for digital image signals representing an image; first framestore for storing a frame of said digital image signals; a second inputfor digital control image signals representing a control image; secondframe store for storing a frame of said digital control image signals;manipulating means for manipulating signals of said first and secondframe stores to achieve a spatial transformation of the image and thecontrol image, wherein said image and said control image undergo thesame spatial transformation; a source of other digital signalsrepresenting an image; combining means for combining digital signalsfrom said first frame store means, after manipulation thereof, if any,by said manipulating means, with said other digital signals representingan image to produce a composed image, said combining means includingcontrol means for varying the proportion of the digital signals fromsaid first store and said other digital signals used in the composedimage, in dependence on said control image after manipulation thereof bythe manipulating means.
 8. An image composition system as in claim 7,wherein said second frame store stores digital signals which have onevalue when the composed image is to consist of signals from said firstframe store, a different value when the composed image is to consist ofsaid other digital signals, and one or more intermediate values when thecomposed image is to consist of a proportion of the signals from saidfirst frame store and of said other digital signals.
 9. A system as inclaim 8, wherein said manipulating means include an interpolator formanipulating information from two or more pixels of an image from thefirst store to produce a pixel of the manipulated image, to therebyensure that picture quality is maintained.
 10. An image compositionsystem as in claim 7 including means for designating part of the saiddigital image signals or said other digital image signals as a priorityarea and to cause the designated part to be output in the composed imagein preference to other image signals for the same area of the composedimage.
 11. An image composition system as claimed in claim 10, includinga further frame store for storing digital signals representing a controlsignal indicative of the shape of said priority area.
 12. A method ofimage composition comprising the steps of:storing first digital signalsrepresenting an image; storing second digital signals representing animage; storing digital signals representing an input control signal;manipulating the first or second digital signals to provide spatialtransformation thereof, and manipulating the control signal to providethereby an output control signal which has undergone the same spatialtransformation as one of the first or second digital signals; andcombining first and second digital signals in dependence on said controlsignal.
 13. An image composition system comprising:a first frame storewhich stores digital signals representing a control image; graphic meansincluding manually operable coordinate selecting means for selectivelyproviding and storing digital control image signals in said first framestore which represent a control image and define at least a selectedarea of an image frame, wherein said selected area has a boundary, saidgraphic means causing the signals in the first frame store to changegradually in value at the boundary of the selected area over a distanceof a plurality of image points; a second frame store; and control meansapplying digital signals representing at least a part of a desired videoimage to said second frame store in proportion determined by themagnitude of respective digital signals stored in said first framestore, said proportion changing gradually in the vicinity of and towardthe boundary of said selected area.
 14. An image composition system asin claim 13 wherein said graphic means generate digital signalsrepresenting at least a part of a frame of said control image.
 15. Animage composition system as in claim 13 wherein said control meansinclude operator-controlled pressure sensitive means which vary theeffect of the digital signals representing said control image on thedigital signals applied to said second frame store, as a function ofoperator-applied pressure.
 16. An image composition system as in claim13 wherein said graphic means comprise means for generating, so as toform part of said image, digital image signals representing a desiredcolor.
 17. An image composition system as in claim 13 in which saidgraphic means include means responsive to a selection by said manuallyoperable coordinate selecting means for providing digital signals for apatch of image points positionally located by said selection, thedigital signals for said patch changing gradually in value from aninterior region toward the edge of the patch and the digital signalswhich are provided for storage in said first frame store beingresponsive thereto, said digital signals which are provided for storagein the first frame store varying at the boundary of the respective areaover the distance of a plurality of image points.
 18. An imagecomposition system as in claim 17 in which said graphic means includemeans for providing digital signals representing intensity values forsaid patch of image points, shape means for providing further signalsrepresenting a desired profile for said patch, means for combining saidintensity value signals respectively with any digital signals stored insaid first frame store for corresponding image points, the proportionsin which said signals are combined being a function of said profilesignals.
 19. An image composition system according to claim 18 whereinsaid shape means are capable of selectively providing further signalsrepresenting different profiles simulating different draftingimplements.